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ABSTRACT 

A meta-analysis examines the predictive validity of 
three methods for evaluating the education and experiences (ES^E) of 
applicants for entry and full-performance level jobs, focusing on 
federal white-collar jobs. The following are methods of rating 
education and experience: (1) Point, assigning points to the duration 
and type of past education and experience; (2) the Knowledge, Skills, 
and Abilities (KSA) method based on job analysis and comparison with 
the applicant's KSA; and (3) the Behavioral Consistency method, also 
based on KSAs from a job analysis, with achievement-based content 
benchmarks. The sample consisted of 1,399 new Federal Government 
hires, representing 10 jobs with a substantial number of selections 
per year. Supervisory ratings of job performance were obtained. The 
Point method was used to select applicants for three jobs, the KSA 
for four, and Behavioral Consistency for three. Moderate support was 
found for the validity of education and experience ratings in 
predicting full-performance level job performance, but not for 
entry-level jobs. The Point method was noL valid for full-performance 
or entry-level jobs. Applicant self-assessment might provide a way of 
measuring the general KSAs required for entry-level trainees' jobs. 
Attachment A is an example of a job performance rating scale, and 
Attachment B summarizes the validity of education and experience 
ratings by job level and rating method. (SLD) 
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Validity of Three E&E Rating Methods for 
Entry and Full Performance Level Jobs 



Today I would like to talk about the predictive validity of three 
methods for evaluating the education and experience of applicants 
for entry and full performance level jobs. This study covers 
white collar jobs in Federal professional, administrative, 
technical, and clerical occupations. Typically the job 
relatedness of these methods has been documented with content 
validity evidence. Although ratings of education and experience 
have been in use for many decades, it has only been within the 
past few years that studies on the criterion-related validity of 
these methods has been reported in the research literature. 

McDaniel, Schmidt and Hunter (1988) conducted a meta-analytic 
study based on an extensiv.j review of published and unpublished 
validity research on ratings of education and experience. A 
typology for classifying methods of rating education and 
experience proposed by Ash (1981) was used to categorize the 
validity coefficients in this meta-analytic study. Three of the 
four categories covered by this meta-analytic research represent 
methods of rating education and experience described in the 
present study. 1 will refer to them as the Point, KSA , and 
Behavioral Consistency methods. 

The Point method includes education and experience ratings that 
assign points to the duration and type of past education and 
experience. Benchmarks describing the type of education or 
experience required for several quality levels is developed by 
individuals familiar with the job. Typically these benchmarks 
are used by personnel ists to evaluate the education and 
experience of applicants that is presented on a general 
application form. In the Federal system points are given for 
several broad quality levels and are often supplemented with 
additional points for other specific job related factors. 
Results from McDaniel 's meta-analytic study do not support the 
validity of the Point method. 

The KSA method describes an approach that is based on ir^portant 
knowledges, skills, and abilities obtained from a job analysis. 
Job applicants are asked to report educcxtion and experience 
that supports each KSA on an oct^Mpat ionally specific supplemental 
form. Benchmarks against wnich to evaluate applicant responses 
are develop. -^d for each KSA by those individuals familiar with the 
occupati-^n. McDaniel's study supports the validity of the KSA 
me thod , 

The Behavioral Consistency method is also based on knowledges, 
skills, and abilities from a job analysis. The knowledges, 
skills, and abilities are combined into several broad achievement 
dimensions (typically about 5). Applicants are asked to describe 
past accomplishments related to each dimension. Applicant 
responses are rated by job knowledge experts and used as 



benchmarks in evaluating applicants. McDaniel found substantial 
support for the validity of the Behavioral Consistency method. 

Although sharing some similarities with the KSA method, the 
Behavioral Consistency approach is distinguished by the content 
of benchmarks used to evaluate rating factors and by the method 
used to scale the value of each benchmark. For the Behavioral 
Consistency method the content of benchmarks is achievement 
based, while for the KSA method the content is education and 
experience based. For the Behavioral Consistency method actual 
applicant achievements are rated by job knowledge experts to form 
a behavioral anchored rating scale for each rating factor. The 
KSA method relies on job knowledge experts to construct education 
and experience benchmarks. 

In a review of alternate predictors of job suc::ess, Hunter and 
Hunter (1984) reported that education, traininv^, and experience 
measures had low mean validities, ranging from .10 to .18, for 
entry-level jobs. These measures are most closely related to the 
Point method of education and experience ratings described in 
this paper. In the same review Hunter reported a mean validity 
of .49 for Behavioral Consistency ratings of education and 
experience used for promotion, when current performance on the 
job was the basis for selection. McDaniel ^s findings could 
explain the difference between these validity coefficients in 
terms of education and experience rating methodology. However, 
it is also possible that the validity of these education and 
experience ratings was influenced by the level at which these 
jobs were f il led. 

The rationale for using education and experience ratings in 
personnel selection is based on the general assumption that past 
job related education, expe^rience, or accomplishments can be used 
to predict future job performance. This suggests that the closer 
these past experiences and accomplishments are to actual job 
performance the hig'her will bo the validity of education and 
experience ratings. Since employers cannot expect applicants for 
entry level jobs to have the specific education or experience 
required to perform the job, it is often difficult to obtain a 
close match between the content of education and experience 
ratings and job requirements. Jt is much easier to identity past 
experience which relates to full performance level jobs. 
Therefore, it is reasonable to hypothesize that ratings of 
education and experience would have higher validity for full 
performance level jobs than for entry-level jobs. A predictive 
validity strategy was used to test this hypothesis. 

In the present study the type of job (e.g., professional, 
administrative, technical, or clerical) and the grade level at 
which it was filled were considered in categorizing jobs as full 
performance or entry-level. Those jobs which do not require 
special job related skills and for which training will take place 



after hiring were categorized as entry level. Those jobs for 
which applicants must have specialized education or experience to 
be able to perform the work with a minimum of training were 
categorized as full performance level. 

New hires for occupations with a substantial number (200) of 
selections per year, from a single examining office, were 
identified as potential candidates for the study. Occupations 
vv'ere selected to sample across grade level and the three methods 
of rating education and experience. The accessibility of 
selection information was also a consideration and the 
occupatjons studied were limited to those covered by examining 
offices in the Washington, DC. area. For those occupations which 
met the above criteria, a review of certification files was made 
to identify those applicants selected for Federal employment. 

The criterion measure selected for this study was supervisory 
ratings of job performance. The rating scales used in the 
criterion measure were a close adaptation of the six Descriptive 
Rating Scales used by the U.S. Employment Service in conducting 
validity studies of the General Aptitude Test Battery (Attachment 
A). The supervisory rating scales were mailed to agency 
personnel officers at the locations where appointments had been 
made. The rating scales were distributed to first line 
supervisors who were asked to rate the job performance of new- 
hires and mail the completed forms back to the research staff. 
The procedure was confidential to protect the privacy of research 
participants and to minimize measurement biases often associated 
with administrative ratings. 

Job performance ratings received from agency supervisors were 
correlated with ratings of education and experience obtained from 
applicant certification files. These validity coefficients were 
corrected for unreliability in the criterion based on research by 
King, Hunter, and Schmidt (1980) that reported a mean interrater 
reliability of » 60 for supervisory ratings of job performance. 
Applicant data was not available from the present study or the 
research literature on education and experience ratings to 
correct for restriction in range. 

Supervisory ratings of job performance were obtained for 1,399 
recent hires representing 10 jobs. Four of the jobs were full 
performance level and six were entry level. The Point method was 
used to select applicants for three jobs, the KSA method for 
four, and the Behavioral Consistency method for three. Although 
the present study did not cover the Point method for f ul 1 
performance level jobs, validity coefficients were obtained from 
the research literature on this job level and method combination. 

Unpublished research by Molyneaux (1953) reported grade level 
information for Federal jobs included in a predictive validity 
study of education and experience ratings based on the Point 
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method . This made it possible to categorize the validity 
coefficients into full performance level and entry level for the 
jobs studied. Thirty validity coefficients were categorized for 
full performance level jobs and 21 for entry level jobs. 

There is little evidence to support the validity of the Point 
method of evaluating education and experience for both the entry 
and full performance level jobs. Validity coef f ic i'^^nts of .02, 
.04 and .09 were obtained for entry level mathematician, 
agricultural management specialist, and cartographer education 
and experience ratings* None of these validity coefficients were 
statistically significant. Mean validity coefficients computed 
from the Molyneaux study were also quite low; .04 for full 
performance level jobs and .13 for entry level jobs. 

Results for the KSA method were mixed. At the full performance 
level statistically significant (p<.05) validity coefficients of 
.41 and .21 were obtained for secretary and dietitian education 
and experience ratings. However, at the entry level for the same 
two occupations, validity coefficients of .06 and .08 were not 
statistically significant. These results suggest that the KSA 
method may only be valid for full performance level jobs. 

Result for the Behavioral Consistency method were consistent with 
those obtained for the KSA method. At the full performance level 
statistically significant (p<.01) validity coefficients of .25 
and .27 were obtained for social worker and pharmacist education 
and experience ratings. At the entry level the validity of Bank 
Examiner education and experience ratings was .04, which was not 
statistically significant. 

Table 1 in the handout summarizes the study results (Attachment 
B). There is no evidence for the validity of the three education 
and experience ratings described in this paper for predicting 
entry level job performance For two methods, KSA and Behavioral 
Consistency, this study found moderate support for the validity 
of educ£ition and experience ratings in predicting full 
performance level job performance. Except for the Point method, 
study results support the hypothesis that ratings of education 
and experience would have higher validity for full performance 
level jobs than for entry level jobs. 

The lack of validity for Point methods of education and 
experience ratings at any job level is not surprising considering 
that these procedures typically use a general application focm 
with credit given for broad types of education and experience. 
This makes it difficult for education and experience obtained 
with the Point method to be closely related to job performance. 
Therefore, the validity of these methods are likely to be quite 
low. This is not a serious problem with Behavioral Consistency 
and KSA methods because they use an occupat ionally specific 
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supplemental form that obtains applicant information related to 
each important job requirement. 



Although the present study indicates that KSA and Behavioral 
Consistency methods of rating education and experience are only 
valid for full performance level jobs, further research is needed 
to confirm these findings with a larger number of occupations in 
a variety of settings. Using methods described in this paper it 
would not be difficult for organizations with centralized 
examining operations to conduct placement follow-up studies of 
new hires identified from certification files. Occupations with 
100 or more selections per year would be good candidates to 
study. Results from such studies could give us the data we need 
to determine how and where education and experience ratings 
should be applied in personnel selection. 

If findings from the present study are confirmed, what do we do 
about the entry level jobs for which education and experience 
ratings have been the most practical examining approach? One 
area that is receiving increasing attention is the use of self- 
assessment techniques in personnel selection. Applicant self- 
assessment might be another way of measuring the general KSA^s 
required for entry level trainee jobs. Although there are 
problems with inflation bias and faking with these procedures, 
they may prove to be promising alternatives to ratings of 
education and experience for situations where other selection 
instruments are not practical to implement. 
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ATTACHMENT A 
Example of Job Performance Rating Scales 



PIACEKENT TOLLjCW-UP SUT^VEY FOR BANK EXAMINER TRAINEE 



Instructions for Ejmployee's Supervisor 

T3ie U. S. Office of Personnel Management (0PM) is conducting a survey of the quality of 
placements fron the Bank Examiner Trainee examination. This information will be used 
only for research purposes to evaluate and iiT5)rove the Federal Government's examining 
program. Tbiese ratings will not effect the eitployee in any way. Responses to this 
survey will be treated cx>nf identially by the 0PM research staff. Nb employee, supervisor 
or organizational unit will be identified in the reporting of survey results. 

Our records show that the employee named below was appointed to a Bank Examiner Trainee 
position in your agency. Your agency has identified you as the enployee's supervisor. 
Please be as accurate and candid as possible in evaluating the job performance of this 
en^^loyee. Use the self -addressed envelope to return the survey form directly to the 
0PM. Your cooperation is greatly appreciated- 



THIS SECTION FOR Cf>M USE 








Social Security Numi^er 




LAST FIRST 


M.I. Agency/location Code 


/ 


(Dnployee's Name) 







Bating Instructions 



Einplqyees vvho have not completed their training period, or who have not been on the job 
or under your supervision long enough for you to know how well they can perform their 
work should not be rated* If this happens to be the case, describe why the employee 
cannot be rated in the space provided at the top of the next page. 

If the enployee is no longer on the job, you should rate factors A through F as well 
as factor G. Do not rate factor G if the employee is still on the job. 

In making ratings, do not let general impressions or some outstanding trait affect your 
judgment. Try to forget your personal feelings about the employee. Rate only on the 
Vvork performed. Fcx:us on one factor at a time. Try to avoid the same ratings on all 
the factors. Only one choice should be checked for each factor. 

For each factor conpare the employee witli other employees on this job that you have 
known. Rate the enployee according to the work that has been done over several weeks 
or months. Do not rate just on the basis of or>e "good" day, or one "bad'Vday, or sane 
single incident. Think in terras of the employee's usual or typical performance. 

Practice and experience usually inprove an employee's skill. However, one employee with 
six months experience may perform better than another with six years experience. Do not 
rate an employee poorer than another merely because of a lesser amount of experience. 

Rate only the factors listed on the rating sheet. Dd not let factors suclri as cocpera- 
tiveness, ability to get along with others, and pronptness influence your ratings. 
Although these characteristics of an en^loyee are important, they are of no value for 
this research study. 
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If you cannot rate this employee, describe the reason why. 



What is the employee's pinesent grade? 

How often do you see this employee in a work situation? 

1. Seldorn 3. Several tiroes a day 

2. Several tiines a week ^4. All the time 

How long have you supervised this enployee? Nuinber of months 



Eating Factors 



A. How much does this enployee get done? (Employee's ability to niake efficient 
use of time and to work rapidly. ) 

Very low work output. Performs only at an unsatisfactory pace. 
Jx>'^ work output. Performs at a slew pace. 
Good wrk output. Performs at an acceptable pace. 
High vork output. Perfonns at a fast pace. 
Very high work output. Performs at an unusually fast pace. 

B. How good is the quality of work? (Eitiployee's ability to do quality work which 
meets acceptable standards. ) 

Quality of wrk is unacceptable and hardly ever meets minimun standards. 
Quality of work is usually acceptable but soinevAiat inferior. 
Quality of work is acceptable but usually not superior. 
Quality of work is usually superior. 
Quality of work is almost always the highest. 

C. How accurate is the work? ( Employee's ability to avoid making mistakes.) 

Makes very many mistakes. Work needs constant checking, 
f^es frequent mistakes. Work needs more checking than is desirable. 
Makes mistakes at times. Work is acceptable ai^id needs only normal checking. 
Makes few mistakes. Vfork seldom needs checking. 
Rarely makes a mistake. Wbrk almost never needs checking. 

D. How much does the eirployee know about the job? (Eniployee's understanding of the 
principles, equipment/ materials, and metiiods that have to do directly or indirectly 
with the workO 



1. Has very limited knowledge. Does not know enough to do the job adequately. 

2. Has little knowledge. Knows enough to get by. 

3. Has acceptable amount of knowledge. Knows enough to do good work. 

4. Has very broad knowledge* Knows enough to do very good work. 

5. Has complete knowledge. Knows enough to perform all work extremely well. 



E. How large a variety of job duties can the einployee perfom efficiently? 
(Ennployee's ability to h ndle several different operations,) 



1* Cannot perfom different operations adequately, 

2. Can perform a limited number of different operations efficiently. 

3. Can perform several different cperations with reasonable efficiency; i.e., 

an acceptable vorker. 
4. Can perform many different operations efficiently. 

5. Can perform an unusually large variety of different operations efficiently. 

F. Considering all the factors already rated, and only these factors, how good is 
this employee? (Summary rating of the enployee's all-around ability to do the job.) 

1. Performance not acceptable. 

2. Performance acceptable at times. 

3. A good and proficient worker. Performance nearly always acceptable. 

4, Perfonrance frequentJ^y exceeds expectations. 

5. An unusually canpetent worker. Peri:onnance almost always exceeds expectations. 



Ccnplete the following only if the worker is no longer on the job. 

G. What do you think is the reason this persoji left the job? (It is not necessary to 
show the official reason if you feel that there is another reason, as this information 
will not be shown to anyone in your organi:z:ation. ) 

1. Terminated because of inability to do the job. 

2. Quit, and I feel that it was because of difficulty doing the job. 

3. Terminated or laid off for reasons other than ability to do the job (absentee- 
ism, reduction in force, etc.) 

4. Quit, and I feel the reason for quitting was not related to ability to do 

the job. 

5. Quit or was prcmoted or reassigned because the worker had learned the job 

well and/or wanted to advance. 



COMPLETEaD BY PERSON RATING WIS EMPLOYEE 




Name 


Date 


Title Series 


Grade 


Mcxiths as a Supervisor in this field 


Months in Goverriment 


Agency/Installation 
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ATTACHMENT B 



Validity of Education and Experience Ratings 
By Job Level and Rating Method 



JOB 


M ^^ 1 1 1^ "t" n 


and Experience Rating Method 








LEVEL 


Point Method 


KSA Mevhod 


BC Method 




iMOiyneauXj i^oo 


Sec '^tary 


Social Worker 




30 jobs 


G.. /7 


QS-9 /II 




2,149 hires 


r = .41 * 


r = .25 * * 


FULL 


Mean r = .04 


( n = 50 ) 


( n = 205 ) 


PERFOR- 








MANCE 




Dietitian 


Pharmacist 






GS-9 


GS-9/11 






r = .21 * 


r = .27 






(n ^ 109 ) 


(n ^ 179 ) 




Cr}'rto<*'rADher 


Secretary 


Bank Examiner 




GS-5/7 


GS-4/5 


GS-5 




r = .09 


r = .06 


r = .04 




( n = 114) 


in = 7 3) 


( n = 286 ) 




Mathematician 


Dietitian 






n Q - f^ / 7 


GS-7 






r = .02 


r = .05 






( n = 127 ) 


( n = 55 ) 




ENTRY 










Ag • Mgmt . Spec . 








GS-5/7 








r = .0 1 








in = 201 ) 








Molyneaux, 19 53 








21 jobs 








1,595 hires 








Mean r = .13 







* p<.0 5 
** p< .01 



